# Instructions for Replication:
To replicate the tables, figures, and numbers in the paper run the following:
- fig_2_bootstrap_ci.py
- sect3_3_ambiguous_docs.py
- sect4_2_3.py
- tables.py
- plots.py
- sect_5_analysis.R

Each script will produce logs and/or plots containing results.

# Data
covid_classified_tweet_ids.csv: The tweet IDs and original labels for the replication example in section 5. Tweets need to be re-hydrated from the Twitter API if you wish to re-classify them.

covid_training_sample.csv: A sample of the training data used to train the original classifier from Block et. al. Used to estimate performance metrics in section 5.

users.csv: Twitter user data and covariates for section 5.

threatmin1.csv: Tweet IDs and labels from the first set of entailment hypotheses.

threatmin2.csv: Tweet IDs and labels from the second set of entailment hypotheses.

trump_test_data.csv: The test set data used in sections 3 and 4.

trump_twitter_supervised.csv: The test set data with labels from supervised classifiers.

trump_test_nli.csv: The test set data with labels from NLI classifiers.

trump_test_in_context.csv: The test set data with labels from in context classifiers.

mcc_matrix.csv: Contains performance estimates and confidence intervals for figure 2.

mcc_matrix_notrump: Contains performance estimates and confidence intervals for figure 2.

trump_train_data.csv: Training data used for the supervised classifiers.

# Model Training and Data Labeling
If you wish to retrain the language models and re-label the documents used in this study I included training data and jupyter notebooks. In addition, you will need the following:
- Access to a GPU with about 24 GB of VRAM for training supervised classifiers and using the NLI and in-context classifiers.
- An account with W&B to conduct the hyperparameter sweep for the supervised classifiers.
- An OpenAI API key for GPT-4 and GPT-3.5

### Model Training Notebooks:

**bow_classifiers_table3.ipynb:** This notebook will retrain the bag-of-words classifiers used for table 3. This notebook does nto require a GPU.

**BERT_classifiers_table3.ipynb:** This notebook will conduct the hyperparameter sweeps, retrain the language models, and reclassify the data for table 3. To train the models you will need access to a GPU. To conduct the hyperparameter sweeps you will also need an account with Weights & Biases.

**nli_classification.ipynb:** This notebook will re-classify the test set using the NLI classifiers. You will need a GPU to run this notebook.

**in_context_classification.ipynb:** This notebook will re-classify the test set using GPT-4, GPT-3.5, and Mistral 7B. To classify the documents with GPT-4 and 3.5 you will need an OpenAI API key. To classify the documents with Mistral 7B you will need a GPU with 24 GB of VRAM.

**trump_topic_classification.ipynb:** This notebook classifies the test set for whether or not documents are related to Trump. These labels are used to produce the numbers in section 3.3. You will need a GPU for this notebook.

# Compute Environment
This research was conducted with the following hardware and software:
### Hardware
CPU: Ryzen 5900x  
GPU: RTX 3090  
RAM: 64 GB  

### Software
OS: Windows 10

Python: 3.11.5  
    pandas: 2.1.1  
    scikit-learn: 1.3.1  
    transformers: 4.35.0  
    tqdm: 4.66.4  
    Tabulate 0.9.0  
    pytorch: 2.0.1  
    pytorch-cuda: 11.8  
    openai: 1.1.1  
    nltk: 3.8.1  
    simpletransformers: 0.64.3  
    datasets: 2.14.5  
    numpy: 1.26.0  
    wandb: 0.15.12  

R: 4.3.1  
ggplot2: 3.4.2  
dplyr: 1.1.2  
mltools: 0.3.5
dotwhisker: 0.8.1  

